Skip to content

MAPREDUCE-7503. Fix ByteBuf leaks in TestShuffleChannelHandler #7500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Mar 14, 2025
Merged

MAPREDUCE-7503. Fix ByteBuf leaks in TestShuffleChannelHandler #7500

merged 2 commits into from
Mar 14, 2025

Conversation

stoty
Copy link
Contributor

@stoty stoty commented Mar 12, 2025 •
edited by slfan1989
Loading

Description of PR

JIRA: MAPREDUCE-7503. Fix ByteBuf leaks in TestShuffleChannelHandler.

Fixes ByteBuf leaks in TestShuffleChannelHandler. For some reason the ByteBuf leaks are not detected on older (8,11,17) JVMs, though in retrospect it's quite obvious that the tests don't release ByteBufs as they should.

How was this patch tested?

The patch was tested on my JDK23 branch.
The test probably cannot be run with JDK23 on trunk.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 35m 21s trunk passed
+1 💚 compile 0m 28s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 27s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 28s trunk passed
+1 💚 mvnsite 0m 31s trunk passed
+1 💚 javadoc 0m 33s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 28s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 49s trunk passed
+1 💚 shadedclient 34m 11s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 18s the patch passed
+1 💚 compile 0m 18s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 18s the patch passed
+1 💚 compile 0m 18s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 0m 18s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 14s /results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚 mvnsite 0m 20s the patch passed
+1 💚 javadoc 0m 19s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 18s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 43s the patch passed
+1 💚 shadedclient 33m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 59s hadoop-mapreduce-client-shuffle in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
114m 47s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7500/1/artifact/out/Dockerfile
GITHUB PR #7500
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2b91e841e013 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e252de3
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7500/1/testReport/
Max. process+thread count 544 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7500/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor

@stoty Overall it looks good. Can we fix the checkstyle issue?

@stoty
fix checkstyle warnings
7384215 
move release methods earlier
@stoty
Copy link
Contributor Author

stoty commented Mar 13, 2025

Thank you @slfan1989 .
I've hepfully fixed the checkstyle issues.
I've also moved the release calls to right after the last access of the potential ByteBuf variable in most cases.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 37m 20s trunk passed
+1 💚 compile 0m 28s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 26s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 28s trunk passed
+1 💚 mvnsite 0m 31s trunk passed
+1 💚 javadoc 0m 33s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 28s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 48s trunk passed
+1 💚 shadedclient 34m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 19s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 19s the patch passed
+1 💚 compile 0m 18s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 0m 18s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 14s the patch passed
+1 💚 mvnsite 0m 21s the patch passed
+1 💚 javadoc 0m 20s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 19s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 43s the patch passed
+1 💚 shadedclient 33m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 1m 2s hadoop-mapreduce-client-shuffle in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
116m 9s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7500/2/artifact/out/Dockerfile
GITHUB PR #7500
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux d36abdd245f8 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 7384215
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7500/2/testReport/
Max. process+thread count 548 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7500/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor

@stoty Thanks for the contribution! Merged into trunk.

@slfan1989
Copy link
Contributor

@stoty I've identified a minor issue, which should be related to MapReduce. I will create a MapReduce Jira to associate with it.

@slfan1989 slfan1989 changed the title HADOOP-19498. Fix ByteBuf leaks in TestShuffleChannelHandler MAPREDUCE-7503. Fix ByteBuf leaks in TestShuffleChannelHandler Mar 14, 2025
@slfan1989 slfan1989 merged commit cd8f18b into apache:trunk Mar 14, 2025
4 checks passed
@stoty
Copy link
Contributor Author

stoty commented Mar 14, 2025

Thank you @slfan1989 .

I think it's quicker to just move the existing JIRA into the new project in these cases.

YanivKunda pushed a commit to YanivKunda/hadoop that referenced this pull request Mar 23, 2025
@stoty @YanivKunda
MAPREDUCE-7503. Fix ByteBuf leaks in TestShuffleChannelHandler (apach…
cfb3fd3 
…e#7500) Contributed by Istvan Toth.

Signed-off-by: Shilun Fan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slfan1989 slfan1989 slfan1989 approved these changes

Assignees
No one assigned
Labels
MapReduce trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@stoty @hadoop-yetus @slfan1989